During the process of operating a motor vehicle, it is necessary for the operator to obtain information concerning the proximity of various dangerous objects and their relative velocities for the operator to may sound driving decisions, such as whether or not there is enough time to change lanes. This information should be obtained from the area that completely surrounds the vehicle. In order to gather this information, the operator is frequently required to physically turn his or her head to check for occupancy of a blind spot, for example. In taking such an action, the attention of the driver is invariably momentarily diverted from control of the vehicle.
For an automobile, the blind spots typically occur on either side of the vehicle starting approximately at the position of the driver and extending backwards sometimes beyond the rear of the vehicle. These blind spots depend heavily on the adjustment of the angle of the mirror. Different areas are in the blind spot depending on the mirror angle. Since it is in general not known whether or how the mirror is set for the particular vehicle, a blind spot detector must detect objects anywhere along the side of the vehicle regardless of the mirror setting.
The problem is more complicated for trucks that not only have much larger blind spots along the sides of the vehicle but also have a serious blind spot starting in front of the right front bumper of the vehicle and extending beyond the right door. This blind spot is particularly serious in urban driving where small vehicles, motorcycles, pedestrians, bicycles etc. in this area can be completely hidden from the view of the driver.
Many systems have been designed which attempt to rotate the mirror to pick up or allow a driver to visually see the object in the blind spot. This is difficult to do without knowledge of the location of the eyes of the driver. For most systems that do not incorporate an occupant sensor capable of determining the location of the driver's eyes, there is a risk that the mirrors will be positioned wrongly thus exacerbating rather than helping the blind spot detection problem. Also, a system that rotates the mirror will make the driver nervous since he or she will not be able to see the scene that he or she is accustomed to seeing in the mirror.
Many systems have also been proposed that display a view of the blind spot, using a video camera, onto a display either on the instrument panel or on the windshield as a “heads-up” display. Any system which displays a picture of the object on the screen that is inside the vehicle is also going to confuse the driver since he or she will not be able to relate that picture to an object such as another vehicle in the blind spot on the side of the host vehicle. Additionally, the state of the art of such displays does not provide equally observable displays at night or in bright sunlight. Thus, displays on a CRT or LCD are not natural and it is difficult for a driver to adjust to these views. The lighting of the views is too faint when sunlight is present and too bright when the vehicle is operating at night. Therefore, none of these television-like displays can replace the actual visual view of the occupant.
Other systems that are based on radar or ultrasound have also not been widely adopted for reasons related to cost, accuracy and false alarms. Both systems use beams of energy that can become several feet in diameter by the time that they reach the edges of the blind spot and thus can confuse a large vehicle or a guardrail, sign, parked car etc. two lanes over with a vehicle in the blind spot. Some such systems attempt to filter threatening objects from non-threatening objects based on the relative speed of the object and thus err by eliminating a significant number of such threats. A tradeoff exists in all such systems where, if all threatening objects are made known to the driver, the false alarm rate becomes unacceptable and the driver soon loses confidence in the system and ignores it. If the false alarm rate is kept low, many dangerous situations are ignored.
Thus, all the prior art systems discussed above have serious failure modes. The lesson is that if a vision-based system such as the rear view mirror is going to be replaced with a non-vision system then the non-vision system must be almost as good as the vision system.
Some other problems come about when a vehicle strays into the lane of the host vehicle. Most systems will fail to warn the operator and thus an accident can result. Thus, the blind spot problem is really two problems relating to the motion of the potentially striking vehicle and the potentially struck vehicle.
The problem that is addressed by this invention is to determine what information is needed about the object that is in the blind spot and then the manner in which this information is presented to the vehicle operator so as to eliminate accidents caused by the failure of the operator to see such an object. This information includes the accurate location of the object relative to the host vehicle, its size, its relative and/or absolute speed, and the identity or kind of object. This information must be known regardless of the changes in road geometry such as steep hills and sharp curves or changes in environmental conditions. Naturally, the system must be low cost if it is going to be purchased by the public.
Studies have shown that giving the driver an extra half-second could eliminate as many as 50 percent of the accidents. Thus, the risk of an accident must also be communicated to the operator in a timely fashion to permit the driver to take evasive action or not take a particular action such as a lane change.
What is needed therefore is a system that acts like the eyes of the driver and interprets the situation and only gives a warning when there is a real possibility of an accident. A passive warning can be given in the form of a light on the mirror whenever an object is in the blind spot but an active signal such as an audible signal or an intervention in the steering of the automobile can only be provided when it is necessary to prevent the accident. This system must work with very high reliability and accuracy since the consequences of an error can be serious injuries or death.
1. Prior Art on Blind Spot Detection Systems
In “A Survey of Automotive Collision Avoidance Systems” by Uttamkumar Dravidam and Sabri Tosunoglu of the Florida International University, the authors provide a good review of the field of obstacle sensors. What follows is a summary of their analysis. Obstacle sensors such as used for blind spot detection can be divided into three types which will now be described:
Optical sensors include passive infrared, laser radar and vision. They generally are sensitive to external environmental conditions, which may not be a problem for blind spot detection since the objects to be detected are usually nearby the host vehicle. Passive infrared and vision cannot provide a direct measurement of distance to an object unless part of the field of view is illuminated by a point or structured light. Laser radar does provide the capability of direct distance measurement, as will be described below, and a stereo camera can also provide distance information.
AMCW (amplitude modulated continuous wave), FMCW (frequency modulated continuous wave) and impulse radar are not generally affected by adverse environmental conditions. Although relatively expensive, FMCW radar is a good technique for long-range distance measurement providing the object to be measured can be sorted out. Radar in general has a high false alarm rate due to the large pixel size at any significant distance from the host vehicle, to multipath effects and reflections off of signs, bridges, guardrails etc.
Ultrasonics is good in applications where only short relative distance measurements are required, because it is able to provide high distance to the target resolution for a relatively low cost. However, for imaging applications, the slow speed and relatively large pixel size renders ultrasonics marginal. Also, ultrasonic waves can be significantly distorted by thermal gradients and wind.
Various researchers have attempted combinations of these technologies with the particular combination of laser radar and pulse or FMCW being quite advantageous for long distance collision avoidance applications.
What follows in a brief description of the principles of operation for different types of sensors including their main advantages and disadvantages. For blind spot applications, sensors should be able to accurately determine the location of the object and the speed of the obstacle relative to the host vehicle. How well this is achieved can be measured with the following indicators:                Sensing range: the maximum and minimum range over which the technique can be used.        Range Resolution: the relative change in range that can be measured.        Pixel Resolution: the width of the beam or size of the pixel received and to which the sensor is sensitive.        Response time: how quickly the sensor can respond to a change in the blind spot occupancy.        Ultrasonics: These sensors work by measuring the time-to-flight of a short burst of ultrasound energy typically at a frequency of 40-200 kHz. The time taken for the ultrasonic waves to travel to and return from the obstacle is directly proportional to the distance between the obstacle and the host vehicle. The main advantage is their relative low cost and small size. These sensors are also very sensitive to changes in the density of air that can be caused by, e.g., high wind velocity and temperature gradients.        Passive Infrared: These sensors measure the thermal energy emitted by objects. Their main advantage is their low cost and small size, and main disadvantage is their inability to determine the distance to a detected object and slow response time.        Laser Radar: As with regular radar, two techniques exist: (1) a pulsed-beam of infrared light coupled with time-of-flight measurements, and (2) the modulation of a continuous light beam. The pulsed technique offers long range, high directionality, and fast response time. Its limitations are its sensitivity to environmental conditions.        FMCW or AMCW Radar: This type of radar uses modulated microwave frequencies, so that the frequency difference between the reflected and the transmitted signal is proportional to the relative velocity of the object. When two waves of slightly different frequencies are used, the distance to the object can also be determined by the phase relationship between the two received reflections. Despite its high cost, this technique offers the advantages of being insensitive to environmental conditions, but the disadvantage of having a large pixel size.        Impulse Radar: This radar differs from FMCW in that it uses very short pulses instead of a continuous wave. Like FMCW radar, it is insensitive to environmental conditions, but the cost is significantly lower than FMCW. Distance can be determined by time-of-flight measurements and velocity can be determined from successive distance measurements. It also has the disadvantage of having a large pixel size resulting in a high false alarm rate and too little information to permit object identification.        Capacitive and Magnetic: Capacitive and magnetic sensors are able to detect close objects (within about 2 m.), using the capacitance or magnetic field variations between electrodes excited at low frequencies, typically about 5 kHz. Despite their limited range, they are low in cost, and robust to external environmental effects. The poor resolution compared to other techniques makes it unlikely that these devices will be used for blind spot detection since most objects are close to the vehicle.        Vision Systems: These techniques are based on the use of a camera and image-processing software. They are sensitive to external environmental conditions; however, this is not a significant shortcoming for blind spot detection.        
Considering now the patent prior art.
U.S. Pat. Nos. 4,766,421; 4,926,170; 5,122,796; 5,311,012; 5,122,796; 5,354,983; 5,418,359; 5,463,384 and 5,675,326 and International Publication No. WO 90/13103 are all assigned to the same firm and all describe a modulated optical system. However, these references do not disclose a camera and in fact, each receiver is a single pixel device. The sensor is not mounted on the side rear view mirror but instead is mounted on the rear of the vehicle. These patents do disclose the use of multiple detectors and thereby achieving a sort of mapping of the detected object into one of several zones. The references also provide a crude velocity measurement of the object moving from one zone to another. Otherwise, they do not provide accurate ranging.
All of these patents describe a blind spot detection system wherein beams of infrared radiation are sent from the interrogating or host vehicle at a significant angle in order to illuminate possible objects in an adjacent lane. No direct measurement of the distance is achieved, however, in some cases multiple detectors are used in such away that when the adjacent detected vehicle is very close to the detector, that is, below the threshold distance, the sensing of the adjacent vehicle is suppressed. In other cases, multiple beams of infrared are used and distance is inferred by the reception of reflected radiation. The detectors are single pixel devices. No attempt is made to image the detected object. Also, no attempt is made to directly measure the location of the detected object.
U.S. Pat. No. 5,008,678 describes a phased array radar system wherein the antenna can be made to conform to the geometry of an edge of the automobile. The locations of the antenna, however, make it difficult to detect many objects in the side blind spots. The particular location and velocity of such objects are also not accurately determined. No image of the device is formed. The device is based on a single pixel having a relatively large size making recognition and identification of the object impossible.
U.S. Pat. No. 5,087,918 describes the use of a combination of two types of radar: dual frequency Doppler radars and frequency modulated continuous wave radar (FMCW). The system provides an indication of the range of the object from the vehicle but does not indicate where in a plane perpendicular to the vehicle the object is located. Also, the system does not apply pattern recognition so that different types of objects in the blind spot can be identified. This patent gives a good description of the limitations of radar systems.
U.S. Pat. No. 5,229,975 describes a method for diagnosing when the system is not operating properly by placing an LED outside the vehicle next to the sensor. This is a single pixel device and thus no imaging or object recognition is possible. Range is not measured directly but through a series of sensors whereby each sensor covers a particular zone. Thus, no accurate range measurement is provided. As the object moves in the blind spot area, it is sensed by a variety of the sensors and the last one to sense it gives a crude indication of the distance.
U.S. Pat. No. 5,235,316 describes an ultrasonic blind spot detecting system that in fact interrogates as much as 200 degrees around the vehicle. It is mounted in place of the conventional mirror and a new side mirror is provided. The ultrasonic sensor rotates until it locates an object and then it causes the mirror to rotate so that the driver can see the object. The patent does not take an image of the threatening object or the object in blind spot. It is a one-pixel device and it does not employ pattern recognition. Additionally, it provides too much information for the driver thus creating the possibility of driver information overload.
U.S. Pat. No. 5,289,321 describes a camera and an LCD display on the instrument panel. The camera views rearward and driver sees the image captured on an LCD. It does not disclose the camera mounted on the rear view mirror. The main problem is that the LCD driver-viewing screen is more likely to confuse than to aid the driver due to its poor dynamic light intensity range and the ability to relate the image to the location and velocity of the object in the blind spot.
U.S. Pat. No. 5,291,261 describes illumination ports at an angle with respect to single pixel receiver ports. Fiber optics are used to transmit the few pixels to a central processing station. There is no direct ranging. Some crude ranging is accomplished since when the object is in certain zones where the projected light overlays the receiving fields, the reflected light can be sensed. It requires multiple locations and cannot be mounted, for example, on the side rearview mirror.
U.S. Pat. No. 5,325,096 uses Doppler radar to determine the presence and relative velocity of an object blind spot. It filters out stationary objects and concentrates only on those objects that have approximately the same velocity as the vehicle. As a result, many objects, such as a high speed passing vehicle, are missed. A light is used to indicate the presence of an occupying item in the blind spot area and an audible alarm is sounded when the turn signal is activated. There is some crude range measurement possible. It is also a single pixel device and thus, no image of the object can be formed. It invariably will miss objects that move rapidly into blind spot. There is no precise ranging. It does not appear that the system can be easily adjusted for vehicles of different length.
U.S. Pat. No. 5,517,196 describes a multi-frequency radar system using Doppler techniques. Stationary objects are filtered out. In fact, the system also only looks at objects that are traveling at approximately the same speed as the host vehicle. It has a good range of 0.25 to 100 feet. Some problems are that this system will interfere with other vehicles having the same system. There appears to be no direct measurement of the object's position, but it does give a good distance resolution of 0.55 feet. This patent also contemplates the use of steering wheel angle and vehicle speed inputs to the system. Even though ultrasonic, infrared and radar are disclosed, it is still a single pixel system. Once again, the system will invariably miss a high-speed vehicle passing on either the right or the left since it is limited to a two mile per hour velocity difference between the blind spot object and the host vehicle. It also appears to be a very expensive system. Another potential problem is that when an especially long truck having the system of this patent is turning, the system would pick up the end of truck and treat it as they object in the blind spot.
U.S. Pat. No. 5,424,952 describes an optical system using cameras wherein distance is measured stereoscopically. Objects that are not in the adjacent lane are ignored. The problems are that no attempt is made to analyze the image or to determine its velocity and therefore, a high false alarm rate can be expected. Although the image is captured, the information is ignored except for its use to determine a stereo distance.
U.S. Pat. No. 5,467,072 describes a phased array radar system that can scan the blind spot as well as all other areas around vehicle. However, the system does not provide an image and therefore no pattern recognition is possible. The 10 degree divergence angle of radar indicates that a single pixel has a diameter of over 3 feet at 20 feet from the radar transmitter which is insufficient resolution to determine the lane that the threatening vehicle is occupying, especially if there is a slight curvature in the road. Such a system is not sufficiently accurate to provide drivers who are attempting to merge into adjacent lines with sufficiently accurate position information to permit a safe merge under heavy traffic without visual contact. Additionally, there is no pattern recognition claimed or even possible with this low resolution device.
U.S. Pat. No. 5,668,539 uses thermal imaging to recognize a car or truck the blind spot. It uses a vibrating element between the field of view containing the blind spot using three lenses thus giving three different locations and a reference field of view that is the road behind the vehicle. The problems with this device are that this system does not know where the infrared rays are coming from. It could be from the sun or from reflections from the wrong lane. The slow cycle time prevents averaging to eliminate errors. At a 60 km per hour passing rate, the vehicle will travel 1.7 m each cycle based on a 10 hertz cycle rate. The patent also mentions that the form of the signal that comes from a vehicle and the blind spot has high frequency associated with it whereas the form of the signal from the road does not. This is an alternate method of discriminating between a vehicle and the road but one that still lacks resolution.
U.S. Pat. No. 5,670,935 describes a camera and a display where the actual images of the vehicle in the blind spot and behind the subject vehicle are displayed on the visual display. Unfortunately, the various figures in the patent that illustrate this phenomena are not accurate and appear to show that the positions of the vehicles relative to the subject vehicle can be visually seen which is not the case. Thus, the invention described in this patent cannot be used for blind spot detection in the manner that is described in this patent since the relative locations of vehicles cannot be determined. Also, no attempt has been made to identify and analyze objects in the blind spot and warn the driver of a pending accident.
U.S. Pat. No. 5,765,116 describes a system wherein a torque is artificially applied to the steering wheel to keep a driver in the center of his lane. This is not a blind spot related patent but this same technique can be used to prevent a driver from attempting to change lanes when there is an object in the blind spot and thus is applicable to this invention.
U.S. Pat. No. 6,038,496 describes a lane boundary finder. It uses a linear array of LEDs plus a linear CCD with a total of 64 pixels in the CCD array. It can be used for blind spot monitoring, although this is not the main purpose of this invention. The CCD array suffers from the problem that, due to its limited dynamic range, it can be overwhelmed by light from the sun, for example, reflected off a vehicle or other surface. Since there is only a linear array of only 64 pixels, no information as to what is in the blind spot can be obtained. In other words, the system knows that something is in the blind spot but does not know what it is or even accurately where it is. Nevertheless, the use of the scanning system disclosed wherein the particular pixel or the beam that is being activated to create a light on a leading or reflecting surface is an important addition to the technology and may also be used with this invention.
International Publication No. WO 95/25322 describes a passive infrared blind spot detector that processes infrared waves based on a crude form of pattern recognition. There is no accurate ranging and there will likely be a high false alarm rate with this system. There is also sometimes a period when the system is unavailable due to changes in ambient conditions such as the start of a rain shower or when the temperature of the road changes due to shading. It is a one-pixel device and therefore does not permit the location of the object in the blind spot to be determined. This device and other similar passive infrared devices will have trouble distinguishing between a small objects such as a motorcycle which is relatively close to the sensor and larger objects such as a truck which are relatively far away, for example in two lanes over. As a result, it will likely falsely indicate that a relatively large object is within a danger zone when in reality the object is at a distance which does not pose a threat.
International Publication No, WO 99/42856 describes a rear of vehicle mounted blind spot detector based on various radar systems. It has the capability of tracking multiple targets and of accurately determining the ranges to the various targets using range-gating techniques. It does not attempt to capture an image of an object in the blind spot or determine the identity of such an object and thus many non-threatening objects will appear to be threatening. Accordingly, the system can be expected to have a high false alarm rate.
In general, the poor resolution of radar systems requires that they use relative velocity as a filter in order to reduce the false alarm rate. As a result, such systems miss a high speed vehicle which is in the blind spot and was not observed approaching the blind spot by the driver. This is a very common occurrence on European superhighways and in the United States on two lane roads.
Thus, none of the prior art described above discloses a method or apparatus of monitoring the blind spot of the vehicle that analyzes an image of one or more objects that occupy the blind spot, identifying them and determining the location and relative velocity of the objects relative to the host vehicle in a manner that permits an accurate warning to be issued to the driver of a potentially dangerous situation.
2. Definitions
Preferred embodiments of the invention are described below and unless specifically noted, it is the applicants' intention that the words and phrases in the specification and claims be given the ordinary and accustomed meaning to those of ordinary skill in the applicable art(s). If the applicants intends any other meaning, they will specifically state they are applying a special meaning to a word or phrase.
Likewise, applicants' use of the word “function” here is not intended to indicate that the applicants seek to invoke the special provisions of 35 U.S.C. §112, sixth paragraph, to define their invention. To the contrary, if applicants wish to invoke the provisions of 35 U.S.C. §112, sixth paragraph, to define their invention, they will specifically set forth in the claims the phrases “means for” or “step for” and a function, without also reciting in that phrase any structure, material or act in support of the function. Moreover, even if applicants invoke the provisions of 35 U.S.C. §112, sixth paragraph, to define their invention, it is the applicants' intention that their inventions not be limited to the specific structure, material or acts that are described in the preferred embodiments herein. Rather, if applicants claim their inventions by specifically invoking the provisions of 35 U.S.C. §112, sixth paragraph, it is nonetheless their intention to cover and include any and all structure, materials or acts that perform the claimed function, along with any and all known or later developed equivalent structures, materials or acts for performing the claimed function.
A “blind spot”, for the purposes of this invention, will include those areas surrounding a vehicle that could contain an object that may not easily be seen by the driver through the various rear view mirrors but which could pose a threat either to the vehicle occupants or to the occupants of the object, or other others such as pedestrians, in the blind spot.
Pattern recognition is commonly used in practicing the instant invention. “Pattern recognition” as used herein will generally mean any system, which processes a signal that is generated by an object, or is modified by interacting with an object, in order to determine which one of a set of classes that the object belongs to. Such a system might determine only that the object is or is not a member of one specified class, or it might attempt to assign the object to one of a larger set of specified classes, or find that it is not a member of any of the classes in the set. The signals processed are generally electrical signals coming from transducers which are sensitive to either acoustic or electromagnetic radiation and, if electromagnetic, they can be either visible light, infrared, ultraviolet or radar or low frequency radiation as used in capacitive or magnetic based sensing systems.
A trainable or a trained pattern recognition system as used herein means a pattern recognition system which is taught various patterns by subjecting the system to a variety of examples. The most successful such system is the neural network. Not all pattern recognition systems are trained systems and not all trained systems are neural networks. Other pattern recognition systems are based on fuzzy logic, Kalman filters, sensor fusion, correlation as well as linear and non-linear regression. Still other pattern recognition systems are hybrids of more than one system such as neural-fuzzy systems.
To “identify” as used herein will mean to determine that the object belongs to a particular set or class. The class may be one containing, for example, all human objects, automobiles, trucks, bicycles, motorcycles, guard rails, trees, etc. depending on the purpose of the system. In the case where a particular vehicle such as a fire engine is to be recognized, the set or class will contain only a single element, i.e., the vehicle to be recognized.
An “occupying item” of a blind spot may be an automobile, truck, motorcycle, pedestrian, bicycle, animal, guard rail, tree, utility pole, as well as many other objects.
In the description herein the term “approaching” when used in connection with the mention of an object or vehicle approaching another will mean the relative motion of the object toward the vehicle having the sensor system. In other words, the coordinate system used in general will be a coordinate system residing in the vehicle. This convention permits a general description to cover all of the cases such as where a moving vehicle approaches a stationary vehicle or where both vehicles are moving.
An “electronic shutter” or “light valve” as used herein will mean any method of controlling the amount of light that can pass through the device based on an electronic signal control of the device.
3. Pattern Recognition Prior Art
Neural networks as the pattern recognition technology are used in several of the implementations of this invention since it makes the monitoring system robust, reliable and practical. The resulting algorithm created by the neural network program is usually only a few hundred lines of code written in the C or C++ computer language. The resulting systems are easy to implement at a low cost, making them practical for automotive applications. The cost of the CCD and CMOS arrays, for example, have been expensive until recently, rendering their use for blind spot systems impractical. Similarly, the implementation of the techniques of the above referenced patents frequently requires expensive microprocessors while the implementation with neural networks and similar trainable pattern recognition technologies permits the use of low cost microprocessors typically costing less than $10 in large quantities.
The present invention preferably uses sophisticated trainable pattern recognition capabilities such as neural networks. Usually the data is preprocessed, as discussed below, using various feature extraction techniques. An example of such a pattern recognition system using neural networks on sonar signals is discussed in two papers by Gorman, R. P. and Sejnowski, T. J. “Analysis of Hidden Units in a Layered Network Trained to Classify Sonar Targets”, Neural Networks, Vol.1. pp 75-89, 1988, and “Learned Classification of Sonar Targets Using a Massively Parallel Network”, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples of feature extraction techniques can be found in U.S. Pat. No. 4,906,940 entitled “Process and Apparatus for the Automatic Detection and Extraction of Features in Images and Displays” to Green et al. Examples of other more advanced and efficient pattern recognition techniques can be found in U.S. Pat. No. 5,390,136 entitled “Artificial Neuron and Method of Using Same and U.S. Pat. No. 5,517,667 entitled “Neural Network and Method of Using Same” to Wang, S. T. Other examples include U.S. Pat. No. 5,235,339 (Morrison et al.), U.S. Pat. No. 5,214,744 (Schweizer et al), U.S. Pat. No. 5,181,254 (Schweizer et al), and U.S. Pat. No. 4,881,270 (Knecht et al). All of the above references are incorporated herein by reference.
4. Optics
Optics can be used in several configurations for monitoring the exterior of an automobile. The receiver can be a CCD or CMOS imager, to receive the emitted or reflected light. A laser can either be used in a scanning mode, or, through the use of a lens, a cone of light can be created which covers a large portion of the object in the blind spot. In these configurations, the light can be accurately controlled to only illuminate particular positions of interest on the vehicle. In the scanning mode, the receiver need only comprise a single or a few active elements while in the case of the cone of light, an array of active elements is needed. The laser system has one additional significant advantage in that the distance to the illuminated object can be determined as disclosed in U.S. Pat. No. 5,653,462, incorporated by reference herein in its entirety to the extent the disclosure of this patent is necessary, e.g., for an understanding of the invention.
In a simpler case, light generated by a non-coherent light emitting diode (LED) device is used to illuminate a desired area. In this case, the area covered is not as accurately controlled and a larger CCD or CMOS array is required. Recently, the cost of CCD and CMOS arrays has dropped substantially with the result that this configuration is now the most cost-effective system for monitoring the blind spot as long as the distance from the transmitter to the objects is not needed. If this distance is required, then a laser system using modulation and phase detection or time-of-flight techniques, a stereographic system, a focusing system, a combined ultrasonic and optic system, or a multiple CCD or CMOS array system as described herein is required.
In a particular implementation, the illuminating light is in the form of a modulated infrared laser light that is scanned in a line that illuminates an object in the blind spot. The reflected light is received by a pin diode after passing through a narrow frequency band notch filter. The diode is a single pixel device but since the direction of the transmitted light is known, the direction of the reflected light is also known. The phase of received light is then compared with the transmitted light. The modulating frequency was selected so that no more than one wavelength of light exists within the blind spot area. The location of the reflecting object can then be determined by the phase difference between the transmitted and reflected light. Although the described system uses a linear scan, it is also possible to use a two-dimensional scan and thereby obtain a three-dimensional map of the blind spot. This can be done using a pin diode as described or the light received by the CMOS array can be monitored on a pixel by pixel basis in a manner similar to the PMD system described in Schwarte, et. al. “New Powerful Sensory Tool in Automotive Safety Systems Based on PMD-Technology”, which is incorporated herein by reference. In this latter case, the entire blind spot area may be flooded with modulated infrared light as described in the paper. On the other hand, it is difficult to overcome the light from natural sources such as the sun by a single floodlight source and therefore a line or even a scanning point source permits better distance measurement using a light source of reasonable intensity.
A mechanical focusing system, such as used on some camera systems can determine the initial position of an object in the blind spot. A distance measuring system based of focusing is described in U.S. Pat. No. 5,193,124 (Subbarao) which can either be used with a mechanical focusing system or with two cameras. Although the Subbarao patent provides a good discussion of the camera focusing art and is therefore incorporated herein by reference, it is a more complicated system than is needed for the practicing the instant invention. In fact, a neural network or optical correlation system can also be used to perform the distance determination based on the two images taken with different camera settings or from two adjacent CCD's and lens having different properties as the cameras disclosed in Subbarao making this technique practical for the purposes of this instant invention. Distance can also be determined by the system described in U.S. Pat. No. 5,003,166 (Girod) by the spreading or defocusing of a pattern of structured light projected onto the object of interest. Distance can also be measured by using time-of-flight measurements of the electromagnetic waves or by multiple CCD or CMOS arrays as is a principle teaching of this invention.
In each of these cases, regardless of the distance measurement system used, a trained pattern recognition system, as defined above, can be used to identify and classify, and in some cases to locate, the illuminated object.
5. Optics and Acoustics
Both laser and non-laser optical systems in general are good at determining the location of objects within the two-dimensional plane of the image and a pulsed laser or continuous modulated radar system in the scanning mode can determine the distance of each part of the image from the receiver by measuring the time-of-flight or by phase measurement. It is also possible to determine distance with the non-laser system by focusing as discussed above, or stereographically if two spaced apart receivers are used and, in some cases the mere location in the field of view can be used to estimate the position of the object in the blind spot, for example. Finally, a recently developed pulsed quantum well diode laser also provides inexpensive distance measurements.
Acoustic systems are additionally quite effective at distance measurements since the relatively low speed of sound permits simple electronic circuits to be designed and minimal microprocessor capability is required. If a coordinate system is used where the z-axis is from the transducer to the object, acoustics are good at measuring z dimensions while simple optical systems using a single CCD are good at measuring x and y dimensions. The combination of acoustics and optics, therefore, permits all three measurements to be made from one location with low cost components as discussed in commonly assigned U.S. Pat. Nos. 5,845,000 and 5,835,613.
One example of a system using these ideas is an optical system that uses natural light coupled with a lens and CCD or CMOS array which receives and displays the image and an analog to digital converter (ADC), or frame grabber, which digitizes the output of the CCD or CMOS and feeds it to an artificial neural network (ANN) or other pattern recognition system for analysis. This system uses an ultrasonic transmitter and receiver for measuring the distances to the objects located in the blind spot. The receiving transducer feeds its data into an ADC and from there, the converted data is directed into the ANN. The same ANN can be used for both systems thereby providing full three-dimensional data for the ANN to analyze. This system, using low cost components, will permit accurate identification and distance measurements not possible by either system acting alone. If a phased array system is added to the acoustic part of the system, the optical part can determine the location of the object and the phased array can direct a narrow beam to the location and determine the distance to the object through time-off-light.
Although the use of ultrasound for distance measurement has many advantages, it also has some drawbacks. First, the speed of sound limits the rate at which the position of the object can be updated. Second, ultrasound waves are diffracted by changes in air density that can occur when thermal gradients are present or when there is a high-speed flow of air past the transducer. Third, the resolution of ultrasound is limited by its wavelength and by the transducers, which are high Q tuned devices. Typically, the resolution of ultrasound is on the order of about 2 to 3 inches. Finally, the fields from ultrasonic transducers are difficult to control so that reflections from unwanted objects or surfaces add noise to the data.
6. The Blind Spot Problem and Solutions
The above review of a prior art blind spot detecting systems illustrates that no existing system is believed to be sufficiently adequate. A fundamental problem is that vehicle operators are familiar with visual systems and inherently distrust all other technology. As soon as the non-visual system gives a false alarm or fails to detect an object in the blind spot, the operator will cease to depend on the system. Theoretically, the best systems would be based on cameras that allow the operator to view all of the blind spots. However, there are no adequate display systems that will appear to the operator to be equivalent to an actual view of the scene. CRTs and LCDs require driver concentration and do not have the dynamic range of lighting that is comparable to the real world. Either the display will be too bright at night or too dim during daylight. Although radar systems can accurately measure distance to an object, they are poor at placing the object in the lateral and vertical coordinates relative to the vehicle and thus create many false alarms.
The simplest system must be able to accurately position the object in the blind spot relative to the host vehicle and inform the driver that a collision potential exists if the driver decides to change lines, for example. This warning must be given to the driver either at a place where he can almost subconsciously observe the warning when he is contemplating a lane change maneuver, or it must provide an audible warning if he attempts to make such a lane change maneuver. Finally such a system might even prevent a driver from executing such a maneuver.
A more sophisticated system would be to provide a simple icon image of the host vehicle and all surrounding vehicles as viewed from above. In this manner, with a simple glance, the driver can determine the location and identity of all objects that are in his blind spot or in the vicinity of the vehicle in any direction. If this display is kept simple, then the problems of visual dynamic range become much less severe. That is, if the driver need only see dark objects on a white background and if the size of these objects is significant, than the display could be viewed both at night and under daylight conditions.
To accomplish these goals, it is necessary to positively locate an object in the blind spot and provide some identification as to what that object is. A driver will respond quite differently if the object is a guard rail or a line of parked cars then he will if it is a Porsche over taking him at 150 kph.
Thus, the requirements of the system are to identify the object and to locate the object relative to the host vehicle. To identify the object preferably requires a pattern recognition system such as neural networks or optical correlation systems. To locate the object preferably requires some means of measuring the distance from the camera to the object. A CMOS camera is quite capable of capturing an image of an object in the blind spot and if the camera is an HDRC camera, then it can perform well under all normal lighting conditions from midnight to bright sunshine especially if minimal illumination is provided on dark nights.
The measurement of the distance to the object can be accomplished many different ways including ultrasonically, using laser radar, FMCW or AMCW radar, and range gated pulsed micropower impulse radar. All of these distance measuring techniques as well as stereographic, focusing, structured light, triangulation and other similar techniques are envisioned for use in this invention.
A low-cost preferred approach of solving the distance measuring problem that is consistent with an HDRC camera system is to project onto the volume of the blind spot a series of infrared light pulses. These pulses are created by an array of laser diodes that are displaced from the camera in such a manner that a pulse of light reflected off of an object in the blind spot will appear on a certain pixel area in the camera field of view and since the location of the transmission of the pulse is known and the location of the camera is known, the distance to the reflecting surface is also known by triangulation. By a judicial choice of transmission angles from the laser diode array, the entire volume of the blind spot can be covered with sufficient accuracy so that no significant object can penetrate the blind spot without creating a reflection and thereby permitting the distance to the object to be determined.
Although a series of pulses from a laser diode array are contemplated, other techniques will also accomplished the same goal, however, at a generally higher cost. For example, a continuous laser beam can be used that would scan the blind spot area in either one or two dimensions. Since the direction of the laser will be known that all times its reflection and excitation of pixels on the CMOS array would permit, once again, an accurate, mapping of the distance to various points on the object in the blind spot to be accomplished. This technique however requires a scanning laser system that in general, although more accurate, would be more expensive than a simple array of LEDs. Once again, the photonic mixing device described above would also provide a three-dimensional image of the contents of the blind spot as would a similar and preferred system described below.
Another technique is to superimpose on the blind spot area a pattern of light commonly referred to as structured light. The source of the structured light must be displaced from the imaging array. By observing characteristics of the reflected pattern, such as the distances between portions of the pattern, the distance to the object can be determined. This system, although common in machine vision applications, requires greater computational resources then the simple LED array described above. Nevertheless, it is a viable approach and envisioned for use in the invention.
Various forms of structured light coupled with other patterns which are either inherent in the lens of the camera or are superimposed mathematically on the image can create what is commonly known as Moiré patterns that also permit the determination of the distance from the camera to the object. In some sophisticated examples, this technique can actually provide the equivalent of topographical maps of the object in the blind spot that would be of value in interpreting or identifying the object. However, these techniques require considerable computational power and are not as cost-effective as the simple LED array described above or a linear scanning LED or laser with a pin diode, or equivalent, receiver as disclosed below.
All of these systems permit differentiation between light that is reflected from the transmitted infrared systems and light that comes from the sunlight, for example. It is quite likely that at certain times certain pixels in the camera will receive infrared radiation that overwhelms the reflection of the infrared sent by the host vehicle system. If this radiation comes from pixels other than those that are expected, then the system will know that the results are erroneous. Thus, the systems described above have the capability of permitting the diagnosis of the data and thereby achieving a high accuracy of the results. If the results do not agree with what is expected, then they are ignored. If that happens over a significant period of time, then the operator of the vehicle is warned that the blind spot detection system is non-operational. Using sophisticated image processing and mathematical techniques, however, it is expected that those periods of non-functionality will be minimal. The vehicle operator however will not be subjected to a false alarm but instead will be told that the system is temporarily non-operational due to excessive sunlight etc. A typical driver can easily relate to this phenomenon and thereby would not lose confidence in the system. The use of a narrow notch filter can significantly improve the separation of the artificially illuminated reflected light from the light reflected from the sun.
Initially, one would assume that the only situation that the driver of a vehicle should be concerned with is if he or she decides that he or she wants to change lanes and after looking into the rear view mirror and not seeing an object in the blind spot, that he then proceeds to change lanes. Unfortunately, the blind spot problem is significantly more complicated. The road may be curved and the lane changing maneuver might be quite easily accomplished, however, based on the geometry of the blind spot detecting system, using prior art systems, the driver is warmed that he cannot execute such a lane change. This may be fallacious in that the vehicle that the system determines is in the blind spot may actually be in a different lane. Under the stress of congested driving conditions, the driver will not tolerate an erroneous message and thereby he might lose confidence in the system.
The identification of the object in the blind spot is important and a significant part of the present invention. All previous blind spot detectors have only indicated that there is a reflection from some object that is near the vehicle that may or may not interfere with the desired intentions of the vehicle operator to change lanes or execute some other maneuver. This is very disquieting to a vehicle operator who was told that something is there but not what that something is. For example, let us say that an operator of a vehicle wished to move that vehicle to the situation where he is partially on the shoulder in order to avoid a vehicle which is intruding onto his lane from the right. All systems today would tell the vehicle operator that he cannot do so. The system described in the present invention would say that there is a guard rail fifteen feet to your left, thereby allowing movement of 10 feet onto the shoulder and thereby avoid the vehicle intruding onto the lane from the right. This is a real world situation, yet all existing blind spot detection systems would give an erroneous answer, or no answer at all, to the vehicle operator.
Future automobile safety systems will be based on differential GPS and centimeter accurate maps of the roadway. The blind spot detector of this invention is an interim step to help eliminate some of the accidents now taking place. The particular geometry of the road is unknown to vehicles today, therefore, a blind spot detection system cannot use information that says, for example, that the road is about to take a sudden curve to the left, in its decision-making function. Nevertheless, this is a real situation and the system for detecting objects in the blind spot should not give erroneous information to be operator that he is about to have collision when the cause of this analysis is based on the assumption that the road will be straight when in fact a strong left turn is taking place. This problem cannot be solved absolutely but if features such as angular position of the steering wheel of the host vehicle are data that can be entered into the system, then these types of situations can become less threatening. A preferred implementation of the present invention uses data from other vehicle sources in the decision making process including the steering wheel angle, vehicle speed etc. and map and location information if available.
In the prior art blind spot detection systems, the inventors have realized that the operator of the vehicle cannot be continuously informed that there is an object in the blind spot. Every driver on the highway during rush hour would otherwise be subjected to a barrage of such warnings. Prior art systems have therefore generally provided an optical warning typically placed as an LED on the rear view mirror and an audible alert sounded when the driver activates the turn signal. Unfortunately, under normal driving conditions only about 70% of drivers use their turn signals as an indication of a lane change. Under stressful congested automobile driving situations, one can expect that that percentage would drop significantly. The driver must be warned when he is about to change lines but the activation of a turn signal is not sufficient.
Various studies have shown that the intentions of a driver can be forecasted based on his activities during a several second period prior to execution of the maneuver. Such systems that monitor the driver and, using neural networks for example, try to forecast a driver's action have been somewhat successful. However, these computationally intensive systems are probably not applicable at this time.
Another method is to provide a simulated rumble strip or vibrating actuation to the steering wheel at such time as the driver elects to redirect the motion of the vehicle based on an object in the blind spot. Whereas a rumble strip type message can be sent to the driver, control of the vehicle cannot be assumed by the system since the road in fact may be executing a sharp curve and taking control of the vehicle might actually cause an accident. The rumble strip method, or similar tactile messaging system, is the preferred approach to informing the driver of a potentially dangerous situation. Initially, a resistance would be applied to the steering wheel when the driver attempts to alter the course of the vehicle. Since the system will not know whether the driver is following a curve in the road or in fact changing lanes, the driver will be able to easily overcome this added resistance but nevertheless, it should indicate to the driver that there is a potential problem. If the driver persists, then a slight to moderate vibration would be applied to the steering wheel. Once again, this would be easily overcome by the driver but nevertheless should serve to positively warn the driver that he or she is about to execute a maneuver that might result in an accident based on the fact that there is an object in his or her blind spot.
A preferred implementation of the instant invention will use a passive optical system for monitoring the presence of objects in the blind spot. Pattern recognition technologies such as neural networks and optical correlation systems will be used to positively identify the object that is in the blind spot. This object may be a pedestrian, bicyclist, motorcyclist, guardrail, animal, automobile, truck, or whatever. The system will be trained or otherwise programmed to inform the operator either optically or orally that such an object appears in the blind spot. It will also inform the driver as to which blind spot contains the object. The system can also inform the driver as to whether this object is moving or stationary in an absolute sense and/or also in relation to the host vehicle. This information can be presented to the operator in a variety of ways. Initially, a light or simple icon can appear on the rear view mirror indicating either that some object exists or that a particular object exists. In more sophisticated systems, an icon representing the object can be placed on a simple icon display which shows the vehicle from an overhead view and an icon which shows the blind spot object and its location. Alternately, an oral annunciation can be provided which tells the driver that, for example, there is a guardrail three feet to his left, or that there is an automobile approaching from the rear in an adjacent lane at a relative speed of 100 kph and is currently 50 feet behind the vehicle. All of these types of warnings can be provided if an identification can be made of the object in the blind spot and an accurate measurement made of the position and velocity of that object relative to the host vehicle. This is an object of this invention.
It can be seen from this description that the system of this invention will inform the driver of the type of object in the blind spot, where it is located specifically, what its velocity is relative to the host vehicle, and in more sophisticated systems, will show graphically an icon showing the object relative to the vehicle from an overhead view which is easily understandable by the driver with a mere glance at the display. Therefore, the system overcomes all of the objections and problems described above with respect to the prior art systems.
The use of passive optical camera systems, such as the HDRC camera, has been discussed and the method of using either neural networks, optical correlation, or other pattern recognition systems has also been discussed that illustrates how, in the present invention, the identity of the object occupying the blind spot will be determined. What follows now is a more detailed discussion of how the position of that object will be determined.
As discussed above, many technologies are available for measuring distance, some more expensive than others. The system described here is believed to be a low cost approach and a preferred embodiment of the invention. This invention is not limited to this technology, however, and any practical distance measuring system is contemplated.
Although simple icon displays are contemplated by this invention, this is due to the lack of sophistication or capability of current display technology. In other words, the dynamic range of light that can be emitted by conventional displays is insufficient to display other than the most simple messages. Technology advances and it is expected that accurate color displays with high dynamic range will be available in the not too distant future. When such displays are available, a more accurate representation of the object in the blind spot even to the point of an actual image might become feasible.
This invention does not generally contemplate the use of rear view mirrors to permit the vehicle operator to actually see the contents of the blind spot. This is because to accurately accomplish this requires knowledge of the position of the eyes of the driver. It is been observed that drivers adjust side rear view mirrors over an extended range that renders the use of the mirror angle unsuitable for determining the position of the driver's eyes. Furthermore, the driver may change position without changing the position of the rear view mirror. Occupant sensing systems are now being deployed on vehicles that have the capability of determining the location of the eyes of a vehicle operator. For those vehicles that contain such a system, the possibility exists not only to automatically adjust the mirror to optimally display the contents of the blind spot, but also to change the orientation of the mirror when some object that the driver should be aware of is in the blind spot. This invention therefore contemplates such activities when occupant sensing systems are placed on vehicles.
For the preferred implementation of the system, the light from laser diodes will actually cross the field of view of the camera. If there is a heavy fog, for example, in the vicinity of the blind spot, then the reflection of the light off of the fog will create an elliptical image on the camera sensor array. This would also be true when heavy rain or heavy snowfall is present. This fact can be used to determine visibility. Observations of visibility conditions of objects in the blind spot even during severe weather conditions has led the inventors of this invention to the conclusion that when the visibility is so poor that the passive optical system using laser diodes described herein is not functioning with sufficient accuracy, that the operator of the vehicle should not be operating the vehicle on the roads and therefore the vehicle operator should be informed that safe travel is not possible. Thus, the use of radar or other technologies to view the blind spot, which is actually quite close to the vehicle, is not necessary since vehicle operation should not be permitted when the visibility is so poor that the object cannot be seen in the blind spot by the system of this invention. Nevertheless, this invention can contribute to safe driving in these conditions, if such driving is attempted, since an indication will be obtained by the system based on the elliptical reflections from the laser diode indicating that the visibility is unacceptable.
There will be conditions when the optical system from the CMOS camera has deteriorated due to contamination on the lens. Similarly, the light emitting laser diodes will emit less light if the lenses are soiled. The system of this invention contemplates a continuous diagnostic feature that will permit sensing of either of these conditions. This can be accomplished in a variety of ways such as a laser diode aimed at the road surface close to the vehicle but within view of the CMOS camera. If the reflection over a period of time is not sufficient, then a warning light will appear on the instrument panel informing the driver that maintenance is required. Naturally, there are many other methods by which a similar diagnostic can be accomplished.
For the embodiment of the invention using triangulation, it is desirable for the laser diodes, scanning laser diode or other light source to be displaced as far as reasonably possible from the camera in order to permit the maximum accuracy for the triangulation calculations. In an automobile, as much as six inches exists from one side of the exterior rear view mirror to the other side. This is marginal. For large trucks, the vertical distance separating the top and bottom of the rear housing can be as much as 24 inches. In both cases, the laser diode would be placed at one extreme and the camera at the other extreme of the mirror housing. An alternate approach is to place the camera on the mirror housing but to place the light source on the vehicle side. Alternately, both the camera and the light source can be placed at appropriate positions on the side of the vehicle. The key is that the direction of the light source should cross field of view of the camera at at least a 10 degree angle.
Since the dots or a line created by the light source will be in the infrared spectrum and the majority of the light coming from objects in the blind spot will be in the optical spectrum, the possibility exists to separate them through the use of an infrared filter which will allow more accurately the determination of the location of the reflection from the laser diode onto the optical array. Such filters can be done either mathematically or through the imposition of a physical filter. However this approach requires a mechanical mechanism to move the filter in and out of the camera field of view. Alternately, to eliminate the need to move the filter, a pin diode or equivalent dedicated receiver can be used to receive the reflected infrared light.
The blind spot problem for trucks is particularly difficult. Trucks experience the same type of blind spot as do automobiles where the blind spot extends the length of the vehicle. However, the truck driver is also unable to see objects that are in another blind spot extending from forward of the front of the vehicle back typically 25 feet. This blind spot has been discussed in greater detail in U.S. Pat. No. 5,463,384 and International Publication No. WO 90/13103. Trucks also have blind spots behind the trailer which are problematic during backup maneuvers. The invention disclosed herein is applicable to all three blind spot situations for trucks, automobiles or other vehicles.
It is noteworthy that some trucks have the capability of automatically rotating the side rear view mirrors based on the relative angle between the cab and the trailer. Such mirror systems are designed so that they maintain their orientation relative to the trailer rather than the cab. The blind spot monitoring system of this invention can make appropriate use of this technology to monitor the space along side of the trailer rather then cab.
Buses also have a blind spot directly in front vehicle and children have been run over by school buses when the driver was not aware that a child was crossing in front of the bus after embarking from the bus. The system of this invention is also applicable for monitoring this blind spot and warning a bus driver that a child is in this blind spot.
Naturally, the images obtained from various locations outside of the vehicle can alternately be achieved by cameras or by fiber-optic systems. This invention is not limited to the physical placement of cameras at particular locations when a fire optic transmission system could be used as well.
The principles of this invention can also be used for other purposes such as intelligent cruise control, speed over ground sensors, parking aids, height sensors for active suspensions, anticipatory crash sensors and obstacle detection systems.
Although a trained neural network is contemplated for the preferred embodiments of this invention, adaptive neural networks are also applicable especially where the host vehicle may be towing different loads that could confuse a static trained system. In this case, part of the system can be made adaptive to adjust to the particular load being pulled by the vehicle.